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Abstract 

Survivable  storage  systems  mask  faults.  A  protocol  family  shifts  the  decision  of  which  types  of  faults  from  implementation  time  to 
data-item  creation  time.  If  desired,  each  data-item  can  be  protected  from  different  types  and  numbers  of  faults  with  changes  only 
to  client-side  logic.  This  paper  presents  proofs  of  the  safety  and  liveness  properties  for  a  family  of  storage  access  protocols  that 
exploit  data  versioning  to  efficiently  provide  consistency  for  erasure-coded  data.  Members  of  the  protocol  family  may  assume  either 
a  synchronous  or  asynchronous  model,  can  tolerate  hybrid  crash-recovery  and  Byzantine  failures  of  storage-nodes,  may  tolerate 
either  crash  or  Byzantine  clients,  and  may  or  may  not  allow  clients  to  perform  repair.  Additional  protocol  family  members  for 
synchronous  systems  under  omission  and  fail-stop  failure  models  of  storage-nodes  are  developed. 
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1  Introduction 


Survivable,  or  fault-tolerant,  storage  systems  proteet  data  by  spreading  it  redundantly  aeross  a  set  of  storage- 
nodes.  In  the  design  of  sueh  systems,  determining  whieh  kinds  of  faults  to  tolerate  and  whieh  timing 
model  to  assume,  are  important  and  diffieult  decisions.  Fault  models  range  from  crash  faults  to  Byzan¬ 
tine  faults  [16]  and  timing  models  range  from  synchronous  to  asynchronous.  These  decisions  affect  the 
access  protocol  employed,  which  can  have  a  major  impact  on  performance.  For  example,  a  system’s  access 
protocol  can  be  designed  to  provide  consistency  under  the  weakest  assumptions  (i.e.,  Byzantine  failures  in 
an  asynchronous  system),  but  this  induces  potentially  unnecessary  performance  costs.  Alternatively,  de¬ 
signers  can  “assume  away”  certain  faults  to  gain  performance.  Traditionally,  the  fault  model  decision  is 
hard-coded  during  the  design  of  the  access  protocol. 

This  traditional  approach  has  two  significant  shortcomings.  First,  it  limits  the  utility  of  the  result¬ 
ing  system — either  the  system  incurs  unnecessary  costs  in  some  environments  or  it  cannot  be  deployed  in 
harsher  environments.  The  natural  consequence  is  distinct  system  implementations  for  each  distinct  fault 
model.  Second,  all  data  stored  in  any  given  system  implementation  must  use  the  same  fault  model,  either 
paying  unnecessary  costs  for  less  critical  data  or  under-protecting  more  critical  data.  For  example,  temporary 
and  easily-recreated  data  incur  the  same  overheads  as  the  most  critical  data. 

In  [9],  we  promote  an  alternative  approach,  in  which  the  decision  of  which  faults  to  tolerate  is  shifted 
from  design  time  to  data-item  creation  time.  This  shift  is  achieved  through  the  use  of  n  family  of  access 
protocols  that  share  a  common  server  implementation  and  client-server  interface.  A  protocol  family  supports 
different  fault  models  in  the  same  way  that  most  access  protocols  support  varied  numbers  of  failures:  by 
simply  changing  the  number  of  storage-nodes  utilized,  and  some  read  and  write  thresholds.  A  protocol 
family  enables  a  given  infrastructure  of  storage-nodes  to  be  used  for  a  mix  of  fault  models  and  number  of 
faults  tolerated,  chosen  independently  for  each  data-item. 

The  protocol  family  covers  a  broad  range  of  fault  model  assumptions  (crash-recovery  vs.  Byzantine 
servers,  crash  vs.  Byzantine  clients,  synchronous  vs.  asynchronous  communication,  client  repairs  of  writes 
vs.  not,  total  number  of  failures)  with  no  changes  to  the  client-server  interface  or  server  implementation.  Pro¬ 
tocol  family  members  are  distinguished  by  choices  enacted  in  client-side  software:  the  number  of  storage- 
nodes  that  are  written  and  the  logic  employed  during  a  read  operation. 

In  this  paper,  we  identify  and  prove  the  safety  and  liveness  properties  that  each  member  of  the  protocol 
family  achieves. 

The  remainder  of  this  paper  is  organized  as  follows.  Section  2  describes  our  protocol  family.  Section  3 
describes  the  mechanisms  employed  by  the  protocol  family  to  protect  against  Byzantine  faults.  Section  4 
details  how  asynchronous  protocol  members  are  realized  within  a  common  software  implementation.  Sec¬ 
tion  5  develops  constraints  (e.g.,  on  the  minimum  number  of  storage-nodes  required)  for  asynchronous 
members.  Sections  6  and  7  respectively  prove  the  safety  and  liveness  properties  of  asynchronous  proto¬ 
col  family  members.  As  well,  the  distinct  safety  and  liveness  properties  of  each  protocol  family  member 
are  identified.  Section  8  extends  the  development  of  asynchronous  protocol  members  into  the  synchronous 
timing  model,  yielding  the  synchronous  protocol  members.  Moreover,  additional  protocol  family  members 
for  synchronous  environments  that  tolerate  omission  failures  and  fail-stop  failures  instead  of  crash-recovery 
failures  are  developed.  Each  distinct  storage-node  failure  model  for  synchronous  protocol  family  mem¬ 
bers  leads  to  distinct  constraints  (e.g.,  on  the  minimum  number  of  storage-nodes  required).  In  Section  9 
synchronous  protocol  members  are  extended  to  take  advantage  of  loosely  synchronized  client  clocks.  Sec¬ 
tion  10  discusses  work  related  to  members  of  the  protocol  family. 
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2  The  protocol  family 


We  describe  each  protocol  family  member  in  terms  of  N  storage-nodes  and  an  arbitrary  number  of  clients. 
There  are  two  types  of  operations  —  reads  and  writes.  Each  read/write  operation  involves  read/write  requests 
from  a  client  to  some  number  of  storage-nodes.  We  assume  that  communication  between  a  client  and  a 
storage-node  is  point-to-point  and  authenticated;  channel  reliability  is  discussed  in  Section  2.1.2. 

At  a  high  level,  the  protocol  proceeds  as  follows.  A  data-item  is  encoded  into  data-fragments;  any 
threshold-based  erasure  code  (e.g.,  replication,  secret  sharing  [26],  information  dispersal  [23],  short  secret 
sharing  [15])  could  be  used.  Logical  timestamps  are  used  to  totally  order  all  write  operations  and  to  iden¬ 
tify  data-fragments  from  the  same  write  operation  across  storage-nodes.  For  each  correct  write,  a  client 
constructs  a  logical  timestamp  that  is  guaranteed  to  be  unique  and  greater  than  that  of  the  latest  complete 
write  (the  complete  write  with  the  highest  timestamp).  A  write  operation  is  complete  once  sufficient  benign 
(non-Byzantine)  storage-nodes  have  executed  write  requests.  The  exact  number  of  storage-nodes  that  must 
execute  a  write  request  for  a  client  to  know  that  its  write  operation  is  complete  differs  among  protocol  mem¬ 
bers.  Storage-nodes  provide  fine-grained  versioning;  a  correcf  storage-node  stores  a  dafa-fragmenf  version 
(indexed  by  logical  fimesfamp)  for  each  wrife  requesf  if  executes. 

To  perform  a  read  operation,  a  clienf  issues  read  requesfs  to  a  sef  of  sforage-nodes.  From  fhe  responses, 
fhe  clienf  idenfifies  fhe  candidate,  which  is  fhe  dafa-fragmenf  version  refurned  wifh  fhe  greafesf  logical 
fimesfamp.  The  read  operation  classifies  fhe  candidafe  as  complete,  incomplete  or  unclassifiable  based  on 
fhe  number  of  read  responses  fhaf  share  fhe  candidafe’s  fimesfamp.  If  fhe  candidafe  is  classified  as  complete, 
fhen  fhe  read  operation  is  complefe;  fhe  value  of  fhe  candidafe  is  refurned.  If  if  is  classified  as  incomplete, 
fhe  candidafe  is  discarded,  anofher  read  phase  is  performed  fo  collecf  previous  versions  of  dafa-fragmenfs, 
and  classificafion  begins  anew;  fhis  sequence  may  be  repeated.  If  fhe  candidate  is  unclassifiable,  members 
of  fhe  profocol  do  one  of  fwo  fhings:  repair  fhe  candidate  or  aborf  fhe  read  operation. 

2.1  Family  members 

Each  member  of  fhe  profocol  family  is  characferized  by  four  paramefers:  fhe  timing  model,  fhe  sforage-node 
failure  model,  fhe  clienf  failure  model,  and  whefher  clienf  repair  is  allowed.  Eighf  profocol  members  resulf 
from  fhe  combination  of  fhese  characferisfics,  each  of  which  supporfs  a  hybrid  failure  model  (crash-recovery 
and  Byzanfine)  of  sforage-nodes. 

2.1.1  Timing  model 

Profocol  family  members  are  eifher  asynchronous  or  synchronous.  Asynchronous  members  rely  on  no 
fimeliness  assumpfions  (i.e.,  no  assumpfions  abouf  message  fransmission  delays  or  execufion  rales).  In 
confrasf,  synchronous  members  assume  known  bounds  on  message  fransmission  delays  befween  correcf 
clienls/slorage-nodes  and  Iheir  execufion  rales. 

2.1.2  Storage-node  failure  model 

Family  members  are  developed  wifh  a  hybrid  sforage-node  failure  model  [27] .  Under  a  Iradilional  hybrid 
failure  model,  up  to  t  sforage-nodes  could  fail,  b  <t  of  which  may  be  Byzantine  faulls;  fhe  remainder  could 
only  crash.  However,  we  consider  a  hybrid  failure  model  for  storage-nodes  lhal  crash  and  recover. 

Firsl,  we  review  fhe  crash-recovery  model  from  Aguilera  el  al.  [1].  In  a  syslem  of  n  processes,  each  pro¬ 
cess  can  be  classified  as  always-up,  evenlually-up,  evenfually-down,  or  unslable.  A  process  fhaf  is  always-up 
never  crashes.  A  process  lhal  is  eventually-up  crashes  al  leasl  once,  bul  Ihere  is  a  time  after  which  if  is  per- 
manenlly  up.  A  process  fhaf  is  eventually-down  crashes  al  leasl  once,  and  Ihere  is  a  lime  afler  which  if  is 
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permanently  down.  A  process  that  is  unstable  crashes  and  recovers  infinitely  many  times.  These  classifica¬ 
tions  are  further  refined:  a  process  is  good  if  if  is  eifher  always-up  or  evenfually-up. 

We  combine  fhe  crash-recovery  model  wifh  fhe  hybrid  failure  model  as  follows.  Up  fo  b  storage-nodes 
may  ever  be  Byzantine;  such  sforage-nodes  do  nol  recover  and  are  nol  good.  There  are  af  leasf  N  —  t  good 
sforage-nodes  (where  b  <t).  A  storage-node  fhaf  is  nol  Byzantine  is  said  lo  be  benign  (i.e.,  benign  sforage- 
nodes  are  eifher  always-up,  evenfually-up,  evenlually-down,  or  unsfable). 

We  assume  fhaf  storage-nodes  have  sfable  sforage  fhaf  persisfs  fhroughouf  fhe  crash  and  recover  pro¬ 
cess.  We  assume  poinf-fo-poinl  aulhenficafed  channels  wifh  properfies  similar  to  fhose  used  by  Aguilera  el 
al.  [1].  In  summary,  channels  do  nol  create  messages  {no  creation),  channels  may  experience  finite  dupli¬ 
cation,  and  channels  are  fair  loss.  The  finite  duplicalion  properly  ensures  fhaf  if  benign  process  p  sends  a 
message  lo  benign  process  q  only  a  finile  number  of  times,  Ihen  q  receives  fhe  message  only  a  finite  number 
of  limes.  The  fair  loss  properly  ensures  fhaf  if  benign  process  p  sends  infinilely  many  messages  lo  good 
process  q,  Ihen  q  receives  infinilely  many  messages  from  p. 

The  timing  model  and  storage-node  failure  model  are  inlerdependenl.  In  an  asynchronous  system, 
storage-node  crashes  are  indistinguishable  from  slow  communication.  In  a  synchronous  system,  storage- 
nodes  lhal  crash  could  be  deleclable  via  limeouls  (i.e.,  Ihe  storage-nodes  could  fail-slop).  However,  in  a 
crash-recovery  failure  model,  Ihe  “facl”  lhal  a  storage-node  has  timed  oul  cannol  be  utilized;  Ihe  timeout 
could  be  from  a  storage-node  that,  in  the  future,  may  respond.  The  crash-recovery  failure  model  is  a  strict 
generalization  of  the  omission  and  crash  failure  models.  Under  less  general  failure  models,  the  lower  bound 
on  the  total  number  of  storage-nodes  required  is  reduced  for  synchronous  protocol  members.  As  such, 
we  consider  synchronous  protocol  members  under  omission  and  fail-stop  storage-node  failure  models  in 
Section  8  . 

2.1.3  Client  failure  model 

Each  member  of  the  protocol  family  tolerates  crash  client  failures  and  may  additionally  tolerate  Byzantine 
client  failures.  We  refer  to  non-Byzantine  clients  as  benign  (i.e.,  benign  clients  are  either  correct  or  crash). 
Crash  failures  during  write  operations  can  result  in  subsequent  read  operations  observing  an  incomplete  or 
unclassifiable  write  operation. 

As  in  any  general  storage  system,  an  authorized  Byzantine  client  can  write  arbitrary  values  to  storage. 
Byzantine  failures  during  write  operations  can  additionally  result  in  a  write  operation  that  lacks  integrity; 
the  decoding  of  different  sets  of  data-fragments  could  lead  to  clients  observing  different  data-items.  Mech¬ 
anisms  for  detecting  any  such  write  operation  performed  by  a  Byzantine  client  are  described  in  Section  3. 
These  mechanisms  successfully  reduce  Byzantine  actions  to  either  being  detectable  or  crash-like,  allowing 
Byzantine  clients  to  be  tolerated  without  any  change  to  the  thresholds. 

The  timing  model  and  client  failure  model  are  interdependent.  In  an  asynchronous  system,  readers 
cannot  distinguish  read-write  concurrency  from  a  crash  failure  during  a  write  operation.  In  a  synchronous 
system,  readers  can  distinguish  read-write  concurrency  from  a  crash  failure  during  a  write  operation  (by 
issuing  multiple  read  requests  separated  in  time  by  the  known  bound  on  write  operation  duration).  However, 
in  this  paper,  we  do  not  consider  synchronous  protocol  members  that  take  advantage  of  this  information. 

2.1.4  Client  repair 

Each  member  of  the  protocol  family  either  allows,  or  does  not  allow,  clients  to  perform  repair.  Repair  enables 
a  client  that  observes  an  unclassifiable  (i.e.,  repairable)  candidate  during  a  read  operation  to  perform  a  write 
operation,  which  ensures  that  the  candidate  is  complete,  before  it  is  returned  (see  Section  4.2.2). 

In  systems  that  differentiate  write  privileges  from  read  privileges,  client  repair  may  not  be  possible. 
Non-repair  protocol  members  allow  read  operations  to  abort.  Reads  can  be  retried  at  either  the  protocol  or 
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Timing  model 

Client  failure 

Repairabllity 

Safety 

Liveness 

Asynchronous 

Crash-only 

Repairable 

Linearizable  [12] 

Wait-free  [10,  14] 

Non-repair 

Linearizable  with  read  aborts  [22] 

Single-client  wait-free 

Byzantine 

Repairable 

Byzantine-operation  linearizable 

Wait-free 

Non-repair 

Byzantine-operation  linearizable 

with  read  aborts 

Single-client  wait-free 

Synchronous 

Crash-only 

Repairable 

Linearizable 

Wait-free 

Non-repair 

Linearizable  with  read  aborts 

Single-client  wait-free 

Byzantine 

Repairable 

Byzantine-operation  linearizable 

Wait-free 

Non-repair 

Byzantine-operation  linearizable 

with  read  aborts 

Single-client  wait-free 

Table  1 :  Safety  and  liveness  properties  of  protocol  family  members.  For  details  on  safety  guarantees  see 
Seetion  6.1.  For  details  on  liveness  guarantees  see  Seetion  7.1. 


applieation  level.  At  the  protoeol  level,  eoneurreney  is  often  visible  in  the  timestamp  histories — an  aborted 
read  eould  be  retried  until  a  stable  set  of  timestamps  is  observed.  Other  possibilities  inelude  requiring 
aetion  by  some  external  agent  or  bloeking  until  a  new  value  is  written  to  the  data-item  (as  in  the  “Listeners” 
protoeol  of  Martin  et  al.  [19]). 

2.2  Protocol  guarantees 

Eaeh  member  of  the  protoeol  family  has  distinet  safety  and  liveness  properties.  In  Table  1,  the  guarantees 
made  by  protoeol  family  members  are  summarized.  Safety  guarantees  are  diseussed  in  Seetion  6.1  and 
Liveness  guarantees  are  diseussed  in  Seetion  7.1.  The  safety  and  liveness  properties  aehieved  are  for  the 
hybrid  erash-reeovery  failure  model  of  storage-nodes.  Sinee  the  hybrid  erash-reeovery  failure  model  is  a 
striet  generalization  of  the  hybrid  omission  failure  model  and  hybrid  erash  failure  model,  these  safety  and 
liveness  properties  hold  in  less  general  storage-node  failure  models.  The  liveness  guarantees  assume  no 
storage  exhaustion  on  storage-nodes. 

3  Mechanisms 

This  seetion  deseribes  meehanisms  employed  for  eneoding  data,  and  preventing  Byzantine  elients  and 
storage-nodes  from  violating  eonsisteney.  We  assume  that  storage-nodes  and  elients  are  eomputationally 
bounded  sueh  that  eryptographie  primitives  ean  be  effeetive.  Speeifieally,  we  make  use  of  eollision-resistant 
hash  funetions.  As  well  we  assume  that  eommunieation  is  authentieated. 

3.1  Erasure  codes 

We  eonsider  only  threshold  erasure  eodes  in  whieh  any  m  of  the  N  eneoded  data-fragments  ean  deeode  the 
data-item.  Moreover,  every  m  data-fragments  ean  be  used  to  deterministieally  generate  the  other  N  —  m 
data-fragments.  Example  threshold  erasure  eodes  are  replieation,  Shamir’s  seeret  sharing  [26],  Rabin’s 
information  dispersal  [23],  and  Krawezyk’s  short  seeret  sharing  [15]. 
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3.2  Data-fragment  integrity 

Byzantine  storage-nodes  ean  eorrupt  their  data-fragments,  whieh  we  informally  refer  to  as  an  “integrity 
fault”.  As  sueh,  it  must  be  possible  to  deteet  and  mask  up  to  b  storage-node  integrity  faults.  Cross  eheek- 
sums  [6]  are  used  to  deteet  eorrupt  data-fragments:  a  eryptographie  hash  of  eaeh  data-fragment  is  eomputed, 
and  the  set  of  N  hashes  are  eoneatenated  to  form  the  cross  checksum  of  the  data-item.  The  eross  eheeksum 
is  stored  with  eaeh  data-fragment,  enabling  eorrupted  data-fragments  to  be  deteeted  by  elients  performing 
read  operations  (see  Seetion  4.2.2). 

3.3  Write  operation  integrity 

Meehanisms  are  required  to  prevent  Byzantine  elients  from  performing  a  write  operation  that  laeks  integrity. 
If  a  Byzantine  elient  generates  random  data-fragments  (rather  than  erasure  eoding  a  data-item  eorreetly), 
then  eaeh  of  the  (^)  subsets  of  data-fragments  eould  “reeover”  a  distinet  data-item.  This  attaek  is  similar  to 
poisonous  writes  for  replieation,  as  deseribed  by  Martin  et  al.  [19].  To  proteet  against  sueh  Byzantine  elient 
aetions,  read  operations  must  only  return  values  that  are  written  eorreetly  (i.e.,  that  are  single-valued).  To 
aehieve  this,  the  eross  eheeksum  meehanism  is  extended  in  three  ways: 

Validating  timestamps.  To  ensure  that  only  a  single  set  of  data-fragments  ean  be  written  at  any  logieal 
time,  the  hash  of  the  eross  eheeksum  is  plaeed  in  the  low  order  bits  of  the  logieal  timestamp.  Note,  the 
hash  is  used  for  spaee-effieieney;  instead,  the  entire  eross  eheeksum  eould  be  plaeed  in  the  low  bits  of  the 
timestamp. 

Storage-node  verification.  On  a  write,  eaeh  storage-node  verifies  its  data-fragment  against  the  eorrespond- 
ing  hash  in  the  eross  eheeksum.  The  storage-node  also  verifies  fhaf  fhe  eross  eheeksum  mafehes  fhe  low- 
order  bifs  of  fhe  validafing  fimesfamp.  A  eorreef  sforage-node  only  exeeufes  wrife  requesfs  for  whieh  bofh 
eheeks  pass.  Thus,  a  Byzanfine  elienf  eannof  make  a  eorreef  sforage-node  appear  Byzanfine — only  Byzan- 
fine  storage-nodes  ean  refurn  unverifiable  responses. 

Validated  cross  checksums.  Combined,  sforage-node  verifieafion  and  validafing  fimesfamps  ensure  fhaf 
fhe  dafa-fragmenfs  being  eonsidered  by  any  read  operafion  were  nof  fabrieafed  by  Byzanfine  sforage-nodes. 
To  ensure  fhaf  fhe  elienf  fhaf  performed  fhe  wrife  operafion  aefed  eorreefly,  fhe  eross  eheeksum  musf  be 
validafed  by  fhe  reader.  For  fhe  reader  to  validafe  fhe  eross  eheeksum,  all  N  dafa-fragmenfs  are  required. 
Given  any  m  dafa-fragmenfs,  fhe  reader  ean  generafe  fhe  full  sef  of  N  dafa-fragmenfs  a  eorreef  elienf  should 
have  wriffen.  The  reader  ean  fhen  eompufe  fhe  “eorreef”  eross  eheeksum  from  fhe  generafed  dafa-fragmenfs. 
If  fhe  generated  eross  eheeksum  does  nof  mafeh  fhe  validated  eross  eheeksum,  fhen  a  Byzanfine  elienf 
performed  fhe  wrife  operafion.  Only  a  single-valued  write  operafion  ean  generafe  a  eross  eheeksum  fhaf  ean 
be  validafed. 

4  The  protocol  family  design 

This  seefion  presenfs  fhe  profoeol  family  design  in  fhe  form  of  pseudo-eode  wifh  supporfing  fexf  for  explana- 
fion.  The  pseudo-eode  relies  on  some  new  terms.  The  symbol,  ts,  denotes  logieal  timestamp  and,  ^candidate, 
denotes  the  logieal  timestamp  of  the  eandidate.  The  set,  {Di, . .  denotes  the  N  data-fragments;  like¬ 

wise,  {^i, . . .  jS'a?}  denotes  the  N  storage-nodes.  A  eross  eheeksum  is  denoted  CC.  The  operator  ‘|’  denotes 
eoneatenation. 

The  symbols,  COMPLETE  and  INCOMPLETE,  used  in  the  read  operation  pseudo-eode,  are  defined  in 
Seefion  5.  The  rules  for  elassifying  an  operafion  as  eomplefe  or  ineomplefe  differ  among  profoeol  family 
members. 
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INITIALIZEO  : 

1 :  /*  Each  member  of  the  history  is  a  ( logical  time,  data  value,  cross  checksum  )  triple  * / 
2:  /*  History  is  stored  in  stable  storage  */ 

3:  History  :=  (0,  _L,  _L) 

RECEIVE_TIME_REQUEST()  : 

1:  SEND(TIME_RESPDNSE,  S,  KAX[History.ts]) 

RECEIVE_WRITE_REqUEST(fi,  D,  CC)  : 

1:  if  (VALIDATE(fi,  D,  CC))  then 
2:  /*  Execute  the  write  request  * / 

3:  History  :=  History  U  {ts,  D,  CC) 

4:  SEND(WRITEJlESPONSE,  S) 

5 :  end  if 

VALIDATE);^,  D,  CC)  : 

1:  if  ((HASH(CC)  ^  ts.Verifier)  QR  (HASH(D)  ^  CC[S]))  then 
2:  return  (FALSE) 

3 :  end  if 

4:  /*  Accept  the  write  request  *  / 

5:  return  (TRUE) 

RECEIVE_READ_LATEST()  : 

1 :  /*  Note,  Latest  is  a  singleton  *  / 

2:  Latest  :=  {X  :  X.ts  =  MAX[History.ts],X  E  History) 

3:  SEND(READJlESPONSE,  S.Lateit) 

RECEIVE_READ_PREVIOUS(fi)  : 

1:  PreHistory  :=  {X  :  X.ts  <ts,XE  History} 

2:  /*  Note,  Latest  is  a  singleton  * / 

3:  Latest  :=  {X  :  X.ts  =  MAX[PreHistory.ts],  X  G  PreHistory) 

4:  SEND(READ  JIESPDNSE,  S,  Latest) 


Figure  1 :  Pseudo-code  for  storage-node  S. 


4.1  Storage-node  design 

Storage-nodes  expose  the  same  interfaee,  regardless  of  the  protoeol  member  being  employed — write  and 
read  requests  for  all  protoeol  members  are  servieed  identieally.  Eaeh  write  request  a  storage-node  exeeutes 
ereates  a  new  version  of  the  data-fragment  (indexed  by  its  logieal  timestamp)  at  the  storage-node  (i.e.,  the 
storage-node  performs  eomprehensive  versioning). 

All  stored  data  is  initialized  to  _L  at  time  0,  and  has  a  eross  eheeksum  of  _L.  The  zero  time,  0,  and  the 
null  value,  _L,  are  well  known  values  whieh  the  elients  understand. 

The  storage-node  pseudo-eode  is  shown  in  Figure  1.  The  History  whieh  eontains  the  version  his¬ 
tory  for  the  data-item  is  kept  in  stable  storage  sueh  that  it  persists  during  a  erash  and  subsequent  reeov- 
ery.  Storage-nodes  validate  write  requests  before  exeeuting  them  (to  proteet  against  Byzantine  elients). 
This  is  performed  by  the  funetion  VALIDATE  ealled  by  RECEIVE_WRITE TIEQUEST.  The  value  returned  by 
RECEIVE_READ .LATEST  and  RECEIVEJIEAD PREVIOUS,  Latest,  is  guaranteed  to  be  unique,  sinee  times¬ 
tamps  are  unique  (i.e.,  two  distinet  write  operations  eannot  have  the  same  timestamp). 

4.2  Client  design 

Clients  do  most  of  the  work,  ineluding  the  exeeution  of  the  eonsisteney  protoeol  and  the  eneoding  and 
deeoding  of  data-items.  The  elient  module  provides  a  bloek-level  interfaee  for  reading  and  writing  to  higher- 
level  software. 

4.2.1  Write  operation 
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WRITE(Z)ata)  : 

1 :  /*  Encode  data,  construct  timestamp,  and  write  data-fragments  * / 

2:  Time  :=  READ_TIMESTAMP() 

3:  {Z)i,...,D]v}  :=ENCODE(Z)c(m) 

4:  CC  ■.=  MAKE_CROSS_CHECKSUM({Z)i , . . .  ,Da,}) 

5:  ts:=  MAKE_TIMESTAMP(r(me, CC) 

6:  DO_WRITE({Z)i,...,Z)w},fi,  CC) 

READ_TIMESTAMP()  : 

1:  ResponseSet  := 

2:  repeat 

3:  for  all  5,-  S  {Si , . .  .,Sn}  \ResponseSet.S  do 

4:  SEND(Si,  TIME_REQUEST) 

5 :  end  for 

6:  if  (POLL_FOR_RESPDNSE()  =  TRUE)  then 

7:  (S,  ts)  :=  RECEIVE_TIMEJlESPONSE() 

8:  if  (S  ^  ResponseSet.S)  then 

9:  ResponseSef.=  ResponseSet  yj  (S,ts) 

10:  end  if 

1 1 :  end  if 

12:  nntil  (\ResponseSet\  =N  —  t) 

13:  return  (nAX[ResponseSet.ts.Time]  +  1) 

MAKE_CROSS_CHECKSUM({Z)i ,...,/)«}): 

1:  for  all  D,  G  {Z)i, . . .  jOm}  do 
2:  Hi  :=  HASH(A) 

3 :  end  for 
4:  CC:=Hi\...\Hn 
5:  retnm  (CC) 

MAKE_TIMESTAMP(T/me,  CC)  : 

1:  ts.Time  .=  Time 

2:  ts. Verifier  :=EASR{CC) 

3:  retnm  (ts) 

D0_WRITE({Z)i,...,Z)]v},  ts,  CC)  : 

1:  ResponseSet  := 

2:  repeat 

3:  for  all  S,  6  (Si , . .  .,Sn}  \ResponseSet.S  do 

4:  SEND(S,-,  WRITE_REqUEST,  ts,  A,  CC) 

5 :  end  for 

6:  if  (P0LL_F0R_RESPDNSE()  =  TRUE)  then 

7:  (S)  :=  RECEIVE_WRITE_RESPONSE() 

8:  if  (S  ^  ResponseSet.S)  then 

9:  Re.sponseSet  :=  ResponseSet  U  (S) 

10:  end  if 

1 1 :  end  if 

12:  until  )\ResponseSet\  =  N  —  t) 

Figure  2:  Client-side  write  operation  pseudo-code. 

The  write  operation  pseudo-eode  is  shown  in  Figure  2.  The  WRITE  operation  eonsists  of  determining 
the  greatest  logieal  timestamp,  eonstrueting  write  requests,  and  issuing  the  requests  to  the  storage-nodes. 

First,  a  timestamp  greater  than,  or  equal  to,  that  of  the  latest  eomplete  write  is  determined  by  the 
READ_TIMESTAMP  funetion  on  line  2  of  WRITE.  Responses  are  eolleeted,  and  the  highest  timestamp  is 
identified,  ineremented,  and  returned. 

In  eaeh  iteration  of  the  loop  whieh  eheeks  the  variable  ResponseSet  in  READ  .TIMESTAMP,  additional 
TIME_REQUEST  messages  are  sent  to  those  storage-nodes  from  whieh  no  response  has  yet  been  reeeived. 
Under  the  erash-reeovery  failure  model,  the  elient  must  repeatedly  send  requests  until  suffieient  responses 
are  reeeived  to  implement  a  reliable  ehannel.  During  eaeh  iteration  of  the  loop,  the  elient  polls  to  deter¬ 
mine  if  any  responses  have  been  reeeived.  Only  a  single  response  from  eaeh  storage-node  is  added  to  the 
ResponseSet.  Onee  N  —  t  responses  are  eolleeted,  the  funetion  returns.  Remember,  there  are  at  least  N  —  t 
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BS.kI>(Repair)  : 

1:  ReadResponseSet  DQ_READ(READ .LATEST,  ±) 

2:  loop 

3:  {CandidateSet,  ^■^candidate)  ■—  C}iOOSE_CMDlDkTE{ReadResponseSet) 

4:  if  {\CandidateSet\  >  COMPLETE)  then 

5:  /*  Complete  candidate:  return  value  */ 

6:  if  {'VkLlDkTE.CkmiDkTE.SET{CandidateSet))  then 

7:  Data  DECODE{CandidateSet) 

8:  return  ((^candidate,  Data)) 

9  :  end  if 

10:  else  if  (\CandidateSet\  >  INCOMPLETE)  then 

11:  /*  Unclassifiable  candidate  found:  repair  or  abort  *  / 

12:  if  (Repair  =  TRUE)  then 

13:  if  (VALIDATE_CANDIDATE_SET(Canrftyate5et)  then 

14:  {Dl , . . .  ,Dw}  :=  GENERATE_FRAGMENTS(Ca«af(rfateSet) 

15.  D0_WRITE({/)i  , .  .  .  ,/Ia?} ,  t^candidatC)  f-Cvalid) 

16:  Data  :=DEC0DE({Di,...,Dw}) 

17:  return  ((^candidate,  Data)) 

18:  end  if 

19:  else 

20:  return  (ABORT) 

21:  end  if 

22:  end  if 

23 :  /*  Incomplete  candidate  or  validation  failed:  loop  again  *  / 

24:  ReadResponseSet  :=  D0_READ(READ_PREVI0US,  f^candidate) 

25 :  end  loop 

D0JlEAD{READ_C0MMAND,  ts)  : 

1:  ResponseSet  := 

2:  repeat 

3:  for  all  S,  6  {Si , . . .  ,Sa(}  \ResponseSet.S  do 

4:  SEND(S,-,  READ.COMMAND,  ts) 

5 :  end  for 

6:  if  (P0LL_F0R_RESP0NSE()  =  TRUE)  then 

7:  (S.  Response)  :=  RECEIVE_READJlESPONSE() 

8:  if  (VALinATE(Response.D,  Response. CC,  Response. ts,  S)  =  TRUE)  then 

9:  if  (READ.COMMAND  ^  READ  PREVIOUS)  DR  Response. ts  <  ts  then 

10:  if  (S  ^  ResponseSet.S)  then 

11:  ResponseSet  :=  ResponseSet  U  (S,  Response) 

12:  end  if 

1 3 :  end  if 

14:  end  if 

15:  end  if 

16:  aat\\  (\ResponseSet\  =  N  —  t) 

17:  return  (ResponseSet) 

VALIDATE(D,  CC,  ts,  S)  : 

1:  if  ((HASH(CC)  ^  ts.Verifier)  DR  (HASH(D)  ^  CC[S]))  then 
2:  return  (FALSE) 

3 :  end  if 
4:  retnm  (TRUE) 

VALIDATE_CANDIDATE_SET(Cani7W£!taSet) 

1 :  if  {ByzantineClients  =  TRUE)  then 

2:  (Di , . . .  ,Djv}  :=  QEMES.A1EJFB.AGmmS(CandidateSet) 

3:  CCvaiid  :=MAKE_CROSS_CHECKSUM({Di,...,Djv}) 

4:  if  (CCvaiid  =  CandidateSet.ee)  then 

5:  return  (TRUE) 

6:  else 

7 :  return  (FALSE) 

8:  end  if 

9:  end  if 

10:  return  (TRUE) 


Figure  3:  Client-side  read  operation  pseudo-code. 


good  storage-nodes  that,  eventually,  will  be  up. 

Next,  the  ENCODE  funetion,  on  line  3,  eneodes  the  data-item  into  N  data-fragments.  Hashes  of  the  data- 
fragments  are  used  to  eonstruet  a  eross  eheeksum  (line  4).  The  funetion  MAKE_TIMESTAMP,  ealled  on  line  5, 
generates  a  logieal  timestamp  for  the  write  operation  by  eombining  the  hash  of  the  eross  eheeksum  and  the 
time  determined  by  READ  .TIMESTAMP. 

Finally,  write  requests  are  issued  to  all  storage-nodes.  Eaeh  storage-node  is  sent  a  speeifie  data- 
fragment,  the  logieal  timestamp,  and  the  eross  eheeksum.  The  write  operation  returns  to  the  issuing  elient 
onee  enough  WRITE  JIESPONSE  replies  are  reeeived  (line  12  of  DO.WRITE). 

4.2.2  Read  operation 

The  pseudo-eode  for  the  read  operation  is  shown  in  Figure  3.  The  read  operation  iteratively  identifies  and 
elassifies  eandidates  until  either  a  eomplete  or  repairable  eandidate  is  found  or  the  operation  aborts  due  to 
insuffieient  information  (only  non-repair  members  ean  abort).  Before  a  repairable  or  eomplete  eandidate  is 
returned,  the  read  operation  validates  its  eorreetness. 

The  read  operation  begins  by  issuing  READJL,ATEST  requests  to  all  storage-nodes  (via  the  DDJIEAD 
funetion).  Eaeh  storage-node  responds  with  the  data-fragment,  logieal  timestamp,  and  eross  eheeksum 
eorresponding  to  the  highest  timestamp  it  has  exeeuted.  The  integrity  of  eaeh  response  is  individually 
validated  by  the  VALIDATE  funetion,  line  8  of  D0_READ.  This  funetion  eheeks  the  eross  eheeksum  against 
the  validating  timestamp  and  the  data-fragment  against  the  appropriate  hash  in  the  eross  eheeksum.  Sinee 
eorreet  storage-nodes  perform  the  same  validation  before  exeeuting  write  requests,  only  responses  from 
Byzantine  storage-nodes  ean  fail  the  reader’s  validation.  A  seeond  type  of  validation  is  performed  on  read 
responses  on  line  9.  Eor  responses  to  READ_PREVIDUS  eommands,  the  logieal  timestamp  is  eheeked  to  ensure 
it  is  strietly  less  than  the  timestamp  speeified  in  the  eommand.  This  eheek  ensures  that  improper  responses 
from  Byzantine  storage-nodes  are  not  ineluded  in  the  response  set.  The  read  operation  does  not  “eount” 
deteetably  Byzantine  responses  towards  the  N  —  t  total  responses  eolleeted  for  the  response  set.  Sinee  N  —  t 
storage-nodes  are  good  (by  assumption),  and  the  Byzantine  storage-node  is  not  good,  this  aetion  is  safe. 

After  suffieient  responses  have  been  reeeived,  a  eandidate  for  elassifieation  is  ehosen.  The  funetion 
CHOOSE.CANDIDATE,  on  line  3  of  READ,  determines  the  eandidate  timestamp,  denoted  ^candidate,  whieh  is  the 
greatest  timestamp  in  the  response  set.  All  data-fragments  that  share  ^candidate  are  identified  and  refurned  as 
fhe  eandidafe  sef.  Af  fhis  poinf,  fhe  eandidafe  sef  eonfains  a  sef  of  dafa-fragmenfs  fhaf  share  a  eommon  eross 
eheeksum  and  logieal  fimesfamp. 

Onee  a  eandidafe  has  been  ehosen,  if  is  elassified  as  eifher  eomplefe,  unelassifiable  (repairable),  or 
ineomplefe.  If  fhe  eandidafe  is  elassified  as  ineomplefe,  a  READ  PREVIOUS  message  is  senf  fo  eaeh  sforage- 
node  wifh  fhe  eandidafe  fimesfamp.  Candidafe  elassifieafion  begins  again  wifh  fhe  new  response  sef. 

If  fhe  eandidafe  is  elassified  as  eomplefe  or  repairable,  fhe  eandidafe  sef  is  eonsfrained  fo  eonfain  suffi- 
eienf  dafa-fragmenfs  (see  Seefion  5)  fo  deeode  fhe  original  dafa-ifem.  Af  fhis  poinf  fhe  eandidafe  is  validafed. 
This  is  done  fhrough  fhe  VALIDATE_CANDIDATE_SET  eall  on  line  6  (for  eomplefe  eandidafes)  or  line  13  (for 
repairable  eandidafes)  of  READ. 

Eor  family  members  fhaf  do  nol  folerafe  Byzanfine  elienfs,  fhis  eall  is  a  no-op  reluming  TRUE. 
Olherwise,  fhe  eandidafe  sef  is  used  fo  generafe  fhe  full  sef  of  dafa-fragmenfs,  as  shown  in  line  2  of 
VALIDATE_CANDIDATE_SET.  A  validafed  eross  eheeksum,  CCvaiid>  is  then  eomputed  from  the  newly  gener¬ 
ated  data-fragments.  The  validated  eross  eheeksum  is  eompared  to  the  eross  eheeksum  of  the  eandidate  set 
(line  4  of  VALIDATE_CANDIDATE_SET).  If  the  eheek  fails,  the  eandidate  was  written  by  a  Byzantine  elient; 
the  eandidate  is  reelassified  as  ineomplete,  and  the  read  operation  eontinues.  If  the  eheek  sueeeeds,  the 
eandidate  was  written  eorreetly  and  the  read  enters  its  final  phase.  Nofe  fhaf  fhis  eheek  eifher  sueeeeds  or 
fails  for  all  eorreef  elienfs,  regardless  of  whieh  storage-nodes  are  represenfed  wifhin  fhe  eandidafe  sef. 
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If  necessary  and  allowed,  repair  is  performed:  write  requests  are  issued  with  the  generated  data- 
fragments,  the  validated  cross  checksum,  and  the  logical  timestamp  (line  15  of  READ).  Storage-nodes  not 
currently  hosting  the  write  execute  the  write  at  the  given  logical  time;  those  already  hosting  the  write  are 
safe  to  ignore  it. 

Finally,  the  function  DECODE  recovers  the  data-item  from  any  m  data-fragments.  The  read  operation 
returns  a  logical  timestamp,  data-item  value  pair.  However,  the  client  likely  only  makes  use  of  the  data-item 
value. 

5  Protocol  constraints 

To  ensure  the  desired  safety  and  liveness  properties  are  achieved,  a  number  of  constraints  must  hold.  For 
each  member  protocol,  N  and  m  are  constrained  with  regard  to  b  and  t  (from  the  hybrid  model  of  storage- 
node  failure).  N  is  the  number  of  storage-nodes  in  the  system  and  m  is  the  “decode”  parameter  of  an  m-of-n 
erasure  code  (note,  n  always  equals  N  in  our  system). 

We  develop  the  protocol  constraints  under  the  asynchronous  timing  model.  Synchronous  protocol 
members  are  considered  in  Section  8. 

5.1  Write  operation  definitions 

We  introduce  the  symbol  Qc  in  the  definition  of  a  complete  write  operation. 

Complete  write  operation:  A  write  operation  is  defined  fo  be  complefe  once  a  fofal  of  Qc  benign 
sforage-nodes  have  execufed  fhe  wrife. 

Clearly,  for  a  wrife  operafion  fo  be  durable. 


t  <  Qc- 


(1) 


5.2  Read  classification  rules 

Recall  fhaf  fhe  candidate  is  fhe  dafa-ifem  version,  relumed  by  a  read  requesl,  wilh  fhe  greafesl  logical 
limeslamp.  The  sel  of  read  responses  lhal  share  fhe  candidate’s  fimeslamp  are  fhe  candidate  set. 

To  classify  a  candidale  as  complete,  a  candidale  sel  of  al  leasl  Qc  benign  sforage-nodes  musl  be  ob¬ 
served.  In  fhe  worsl  case,  al  mosl  b  members  of  fhe  candidale  sel  may  be  Byzantine,  Ihus, 

\CandidateSet\  —b>  Qc,  so  COMPLETE  =  Qc  +  b.  (2) 

To  classify  a  candidale  as  incomplefe,  fhe  candidale  musl  be  incomplete  (i.e.,  fewer  lhan  Qc  benign 
storage-nodes  have  executed  Ihe  write).  We  consider  a  rule  for  classifying  incomplete  writes  lhal  lakes 
advantage  of  N  —  t  responses  from  storage-nodes.  In  Ihe  crash-recovery  model,  evenlually,  a  clienl  is 
guaranteed  to  receive  Ihis  many  responses — even  Ihough,  Ihere  may  be  periods  during  which  more  lhan 
t  storage-nodes  are  crashed.  Moreover,  in  an  asynchronous  timing  model,  a  clienl  cannol  expecl  more  lhan 
Ihis  many  responses,  since  up  to  t  storage-nodes  may  never  recover.  The  general  rule  for  classifying  a 
candidate  incomplete  is, 

I CandidateSet\  +t  <Qc,  so  INCOMPLETE  =  Qc-t.  (3) 

Basically,  each  storage-node  lhal  does  nol  respond  musl  be  assumed  to  be  benign,  and  to  have  executed  Ihe 
write  operation. 

There  are  candidates  lhal  cannol  be  classified  as  complete  or  incomplete.  These  candidates  are  termed 
unclassifiable/repairable.  Repairable  protocol  family  members  initiate  repair  of  such  candidates,  Ihus  com¬ 
pleting  Ihem.  Non-repair  protocol  family  members  aborl  upon  encountering  unclassiliable  candidates. 
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Protocol 

Repairable 

Non-repair 

N 

2t  +  2b+l<N 

3f  +  3fo+I  <  A 

Qc 

t  +  b+l<Qc 

t  +  b+\<Qc 

Qc<N~t-b 

Qc<N-2t-2b 

m 

1  <m<  Qc  —  t 

\  <m<  Qc  -\-  b 

Table  2:  Protocol  family  constraint  summary 


5.3  Protocol  properties 

This  section  develops  two  sets  of  constraints:  one  for  repairable  and  one  for  non-repair  protocol  members. 
These  constraints  hold  for  asynchronous  protocol  members  under  a  hybrid  crash-recovery-Byzantine  model 
of  storage-node  failure  with  Byzantine  clients.  The  constraints  do  not  change  if  clients  only  crash.  A 
summary  of  the  constraints  for  the  protocol  family  is  presented  in  Table  2.  Constraints  for  each  protocol 
family  member  are  derived  to  satisfy  a  number  of  desired  properties: 

Write  completion:  This  property  ensures  that  write  operations  by  correct  clients  can  complete. 

Real  unclassifiable/repairable  candidates:  This  property  ensures  that  colluding  Byzantine 
storage-nodes  are  unable  to  fabricate  a  candidate  that  a  correct  client  deems  unclassifiable/repairable  or 
complete. 

Classifiable  complete  candidates:  This  property  is  only  relevant  for  non-repair  protocol  members; 
it  ensures  that  Byzantine  storage-nodes  cannot  make  all  read  operations  abort.  Consider  an  isolated  correct 
client  that  performs  a  write  operation  and  then  a  read  operation  (i.e.,  the  client  does  not  fail  and  there  is 
no  concurrency).  This  property  ensures  that  the  read  operation  will  return  the  value  written  by  the  write 
operation  regardless  of  storage-node  failures. 

Decodable  CANDIDATES:  m  must  be  constrained  so  that  complete  candidates  can  be  decoded.  Moreover, 
m  must  be  constrained  further  for  repairable  protocol  members  so  that  repairable  candidates  can  be  decoded. 

5.4  Repairable  constraints 

Write  completion:  There  must  be  sufficient  good  storage-nodes  in  the  system  for  a  write  operation  by 
a  correct  client  to  complete.  A  client  must  terminate  after  it  receives  N  —  t  responses.  As  well,  up  to  b 
responses  may  be  Byzantine.  Thus,  for  the  write  operation  to  complete  (i.e.,  for  Qc  benign  storage-nodes  to 
execute  write  requests). 


Qc<N-t-b.  (4) 

Real  unclassifiable/repairable  candidates:  To  ensure  that  Byzantine  storage-nodes  cannot  fab¬ 
ricate  an  unclassifiable/repairable  candidafe,  a  candidate  set  of  size  b  must  be  classifiable  as  incomplefe. 
Substituting  \CandidateSet\  =  b  into  (3), 

b  -\-t  <i  Qc-  (5) 

Decodable  repairable  candidates:  Any  repairable  candidate  must  be  decodable.  The  classification 

rule  for  incomplete  candidates  (cf.  (3))  gives  the  upper  bound  on  m,  since  a  candidate  that  is  not  incomplete 
must  be  repairable: 

\  <m<Qc-t.  (6) 
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Constraint  summary: 


Qc  +  t  +  b<N-, 
t  +  b  +  l  <  Qc  —  t  —  b; 
I  <m<  Qc  —  t. 


5.5  Non-repair  constraints 

Some  of  the  repairable  eonstraints  apply  to  non-repair  protoeol  members.  Both  the  write  eompletion  and 
real  unelassifiable/repairable  eandidates  eonstraints  hold  (eonstraints  (4)  and  (5),  respeetively).  However, 
the  write  eompletion  eonstraint  is  supereeded  by  the  elassifiable  eomplete  eandidates  eonstraint. 
Classifiable  complete  candidates:  For  this  property  to  hold,  a  read  operation  must  observe  at  least 
Qc  +  b  responses  from  benign  storage-nodes — suffieient  responses  to  elassify  the  eandidate  as  eomplete  (ef. 
(2)).  A  write  operation  by  a  eorreet  elient  may  only  eomplete  at  N  —  t  storage-nodes  (due  to  asynehrony 
or  benign  erashes);  a  subsequent  read  operation  may  not  observe  responses  from  t  benign  storage-nodes 
(again,  due  to  asynehrony  or  benign  erashes).  These  sets  of  t  storage-nodes  that  do  not  respond  to  the  write 
operation  and  subsequent  read  operation  eould  be  disjoint  sets  (sinee  storage-nodes  ean  erash,  then  reeover, 
or  beeause  of  asynehrony).  Further,  b  observed  responses  may  be  Byzantine.  So, 

Qc  +  b<{{N-t)-t)-b, 

Qc<N-2t-2b.  (7) 

Decodable  complete  CANDIDATES:  A  eandidate  elassified  as  eomplete,  (2),  must  be  deeodable: 

1  <  m  <  Qc  +  b.  (8) 

The  upper  bound  on  m  is  greater  than  Qc,  even  though  Qc  defines  a  eomplete  write  operation.  The  eounter- 
intuitive  upper  bound  on  m  follows  from  the  faet  that  a  write  operation  that  is  eomplete,  may  beeome  unelas- 
sifiable  due  to  storage-node  failures.  Given  this,  only  when  a  write  operation  is  eomplete  and  elassifiable 
as  sueh,  need  it  be  deeodable.  However,  there  is  some  praetieal  value  to  eonstraining  m  <  Qc-  In  a  system 
with  failed  storage-nodes,  a  system  administrator  eould  make  the  judgement  eall  that  a  write  operation  is 
eomplete — so  long  as  m  <  2c  the  system  administrator  eould  then  foree  the  reeonstruetion  of  the  data. 
Constraint  summary: 


Qc+2t  +  2b<N-, 

t  +  b  +  l  <  Qc  +N  —  2t  —  2b; 
I  <m<  Qc  +  b. 


6  Proof  of  safety 

This  seetion  presents  a  proof  that  our  protoeol  implements  linearizability  [12]  as  adapted  appropriately  for 
a  fault  model  admitting  operations  by  Byzantine  elients. 

6.1  Safety  guarantees 

Intuitively,  linearizability  requires  that  eaeh  read  operation  return  a  value  eonsistent  with  some  exeeution 
in  whieh  eaeh  read  and  write  is  performed  at  a  distinet  point  in  time  between  when  the  elient  invokes  the 
operation  and  when  the  operation  returns.  The  adaptations  neeessary  to  reasonably  interpret  linearizability 


12 


in  our  context  arise  from  the  fact  that  Byzantine  clients  need  not  follow  the  read  and  write  protocols  and  that 
read  operations  may  abort  in  non-repair  member  protocols.  We  consider  four  distinct  safety  guarantees: 

Linearizability.  Repairable  protocol  members  with  crash-only  clients  achieve  linearizability  as  originally 
defined  by  Herlihy  and  Wing  [12]. 

Byzantine-operation  linearizability.  Read  operations  by  Byzantine  clients  are  excluded  from  the  set  of 
linearizable  operations.  Write  operations  are  only  included  if  they  are  well-formed  (i.e.,  if  they  are  single¬ 
valued  as  in  Section  3). 

Write  operations  by  Byzantine  clients  do  not  have  a  well-defined  sfarf  time.  Such  operafions  are  con- 
currenf  fo  all  operations  fhaf  begin  before  fhey  complefe  and  fo  all  operations  fhaf  are  also  performed  by 
Byzanfine  clienfs.  A  Byzantine  clienf  can  wrife  “back  in  lime”  by  using  a  lower  logical  limesfamp  fhan  a 
benign  clienf  would  have  used.  Since  wrife  operations  by  Byzanfine  clienfs  are  concurrenl  fo  all  operations 
fhaf  sfarfed  before  if  complefed,  fhey  can  be  linearized  jusf  prior  fo  some  concurrenl  wrife  operation  (if  Ihere 
is  one).  Such  a  linearization  ensures  fhaf  Ihe  Byzantine  “back  in  lime”  wrife  operalion  has  no  effecl  since 
Ihe  value  wrillen  is  never  relumed  by  a  read  operalion. 

In  summary,  Ihere  are  Iwo  types  of  Byzanfine  write  operafions  fhaf  are  of  concern:  wriles  fhaf  are  nol 
well-formed  and  “back  in  lime”  wriles.  In  Ihe  case  fhaf  Ihe  Byzanfine  wrife  operalion  is  nol  well-formed, 
read  operafions  by  benign  clienfs  exclude  if  from  Ihe  sel  of  linearized  operafions.  In  Ihe  case  fhaf  Ihe 
Byzanfine  wrife  operalion  is  “back  in  time”,  Ihe  protocol  family  achieves  somelhing  similar,  in  fhaf  such 
Byzanfine  wrife  operafions  are  linearized  so  fhaf  fhey  have  no  effecl. 

Linearizability  with  read  aborts.  Non-repair  protocol  members  allow  reads  to  abort  due  to  insufficient 
classification  information:  aborted  reads  are  excluded  from  the  set  of  linearizable  operations.  Such  members 
achieve  “linearizability  with  read  aborts”,  which  is  similar  to  Pierce’s  “pseudo-atomic  consistency”  [22]. 
That  is,  the  set  of  all  write  operations  and  all  complete  read  operations  is  linearizable. 

Byzantine-operation  linearizability  with  read  aborts.  For  non-repair  protocol  members  that  tolerate 
Byzantine  clients,  the  safety  property  is  the  combination  of  Byzantine-operation  linearizability  and  lin¬ 
earizability  with  read  aborts. 

6.2  Proof 

Because  return  values  of  reads  by  Byzantine  clients  obviously  need  not  comply  with  any  correctness  criteria, 
we  disregard  read  operations  by  Byzantine  clients  in  reasoning  about  linearizability,  and  define  the  duration 
of  reads  only  for  those  executed  by  benign  clients  only. 

Definition  1  A  read  operation  executed  by  a  benign  client  begins  when  the  client  invokes  READ  locally. 
A  read  operation  executed  by  a  benign  client  completes  when  this  invocation  returns  {timestamp,  value) .  A 
read  operation  by  a  benign  client  that  crashes  before  the  read  completes,  does  not  complete. 

Before  defining  the  duration  of  write  operations,  it  is  necessary  to  define  what  it  means  for  a  storage- 
node  to  accept  and  then  execute  a  write  request. 

Definition  2  Storage-node  S,  accepts  a  write  request  with  data-fragment  D,  cross  checksum  CC,  and 
timestamp  ts  upon  successful  return  of  the  function  VALIDATE(f5,  D,  CC)  at  the  storage-node. 

Definition  3  Storage-node  S,  executes  a  write  request  once  the  write  request  is  accepted.  An  executed 
write  request  is  stored  in  stable  storage. 

It  is  not  well  defined  when  a  write  operation  by  a  Byzantine  client  begins.  Therefore,  we  settle  for 
merely  a  definition  of  when  writes  by  Byzantine  clients  complete. 
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Definition  4  A  write  operation  with  timestamp  ts  completes  onee  2c  benign  storage-nodes  have  exeeuted 
write  requests  with  timestamp  ts. 

In  faet,  Definition  4  applies  to  write  operations  by  benign  elients  as  well  as  “write  operations”  by 
Byzantine  elients.  In  this  seetion,  we  use  the  label  Wts  as  a  shorthand  for  the  write  operation  with  timestamp 
ts.  In  eontrast  to  Definition  4,  Definition  5  applies  only  to  write  operations  by  benign  elients. 

Definition  5  Wts  begins  when  a  benign  elient  invokes  the  WRITE  operation  loeally  that  issues  a  write 
request  bearing  timestamp  ts. 

Lemma  1  Let  ci  and ci  be  benign  clients.  Ifci  performs  a  read  operation  that  returns  {tsi,  vi ),  C2 performs 
a  read  operation  that  returns  {tS2,V2),  and  tsi  =  tS2,  then  vi  =  V2. 

Proof.  Sinee  ts\  =  tS2,  eaeh  read  operation  eonsiders  the  same  verifier.  Sinee  eaeh  read  operation 
eonsiders  the  same  verifier,  eaeh  read  operation  eonsiders  the  same  eross  eheeksum  (remember,  a  eollision 
resistant  hash  funetion  is  employed).  A  read  operation  does  not  return  a  value  unless  the  eross  eheeksum 
is  valid  and  there  are  more  than  b  read  responses  with  the  timestamp  (sinee  only  eandidates  elassified  as 
repairable  or  eomplete  are  eonsidered).  Thus,  only  a  set  of  data-fragments  resulting  from  the  erasure- 
eoding  of  the  same  data-item  that  are  issued  as  write  requests  with  the  same  timestamp  ean  validate  a  eross 
eheeksum.  As  sueh,  vi  and  V2  must  be  the  same.  □ 

Let  Vts  denote  the  value  written  by  Wts  whieh,  by  Lemma  1,  is  well-defined.  We  use  rts  fo  denote  a  read 
operation  by  a  benign  elient  that  returns  {ts,Vts). 

Definition  6  Let  oi  denote  an  operation  that  eompletes  (a  read  operation  by  a  benign  elient,  or  a  write 
operation),  and  let  02  denote  an  operation  that  begins  (a  read  or  write  by  a  benign  elient).  oi  precedes  02  if 
oi  eompletes  before  02  begins.  The  preeedenee  relation  is  written  as  oi  -^02.  Operation  02  is  said  to  follow, 
or  to  be  subsequent  to,  operation  oi. 

Lemma  2  Ifwts  then  ts  <  ts'. 

Proof:  A  eomplete  write  operation  exeeutes  at  at  least  Qc  benign  storage-nodes  (ef.  Definition  4). 
Sinee  Wts  Wts',  the  READ_TIMESTAMP  funetion  for  Wts  eolleets  A  — t  TIME_RESPONSE  messages,  and  so  Wts' 
observes  at  least  b  +  \  TIME_RESPONSE  messages  from  benign  storage-nodes  that  exeeuted  Wts  (remember, 
t  +  b  <Qc^or  all  asynehronous  protoeol  family  members).  As  sueh,  Wts'  observes  some  timestamp  greater 
than  or  equal  to  ts  and  eonstruets  ts'  to  be  greater  than  ts.  A  Byzantine  storage-node  ean  return  a  logieal 
timestamp  greater  than  that  of  the  preeeding  write  operation;  however,  this  still  advanees  logieal  time  and 
Lemma  2  holds.  □ 

Observation  1  Timestamp  order  is  a  total  order  on  write  operations.  The  timestamps  of  write  operations 
by  benign  elients  respeet  the  preeedenee  order  among  writes. 

Lemma  3  If  some  read  operation  by  a  benign  client  returns  {ts,Vts),  with  Vts  7^  -L,  then  Wts  is  complete. 

Proof:  For  a  read  operation  to  return  value  v^,  the  value  must  have  been  observed  at  at  least  Qc  +  b 
storage-nodes  (given  the  eomplete  elassifieation  rule  for  eandidate  sets).  Sinee,  at  most  b  storage-nodes  are 
Byzantine,  the  write  operation  Wts  has  been  exeeuted  by  at  least  Qc  benign  storage-nodes.  By  definition, 
is  eomplete.  □ 

Observation  2  The  read  operation  from  Lemma  3  eould  have  performed  repair  before  returning.  In  a 
repairable  protoeol  member,  a  eandidate  that  is  neither  elassifiable  as  ineomplete  or  eomplete  is  repaired. 
Onee  repaired,  the  eandidate  is  eomplete. 
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Definition  7  wts  is  well-formed  if  ts. Verifier  equals  the  hash  of  eross  eheeksum  CC,  and  for  all  i  G 
,  A^},  hash  CC[i\  of  the  eross  eheeksum  equals  the  hash  of  data-fragment  i  that  results  from  the  erasure- 
eneoding  of  Vts- 

Lemma  4  Ifwts  is  well-formed,  and  ifwts  rfs>,  then  ts  <  ts'. 

Proof.  Sinee  Wts  is  well-formed  it  ean  be  returned  by  a  read  operation.  By  Lemma  3,  read  operations 
only  return  values  from  eomplete  write  operations.  As  sueh,  r,/  must  either  return  the  value  with  timestamp 
ts  or  a  value  with  a  greater  timestamp.  Therefore,  ts  <  ts' .  □ 

Observation  3  It  follows  from  Lemma  4  that  for  any  read  either  Wts  rts  and  Wts  is  the  latest  eomplete 
write  that  preeedes  rts,  or  Wts  A  Oi  and  rts  A  ^ts  (i-o.,  Wts  and  rts  are  eoneurrent). 

Observation  4  It  also  follows  from  Lemmas  3  and  4  that  if  rts  rts',  then  ts  <  ts' .  As  sueh,  there  is  a 
partial  order  ^  on  read  operations  by  benign  elients  defined  by  the  timestamps  assoeiated  with  the  values 
returned  (i.e.,  of  the  write  operations  read).  More  formally,  rts  fts'  ts  <  ts'. 

Sinee  Lemma  2  ensures  a  total  order  on  write  operations,  ordering  reads  aeeording  to  the  timestamps  of 
the  write  operations  whose  values  they  return  yields  a  partial  order  on  read  operations.  Lemma  4  ensures  that 
this  partial  order  is  eonsistent  with  preeedenee  among  reads.  Therefore,  any  way  of  extending  this  partial 
order  to  a  total  order  yields  an  ordering  of  reads  that  is  eonsistent  with  preeedenee  among  reads.  Thus, 
Lemmas  2  and  4  guarantee  that  this  totally  ordered  set  of  operations  is  eonsistent  with  preeedenee.  This 
implies  the  natural  extension  of  linearizability  to  our  fault  model  (i.e.,  ignoring  reads  by  Byzantine  elients 
and  the  begin  time  of  writes  by  Byzantine  elients);  in  partieular,  it  implies  linearizability  as  originally  defined 
by  Herlihy  [12]  for  repairable  profoeol  family  members  if  all  elienfs  are  benign. 

Observation  5  Nole,  Lemma  4  does  nol  address  reads  fhaf  aborf  (if  only  addresses  reads  fhaf  refurn  a 
value).  Read  operations  fhaf  aborf  are  exeluded  from  fhe  sef  of  operations  fhaf  are  linearized. 

7  Proof  of  liveness 

This  seefion  presenfs  fhe  proof  of  fhe  liveness  properfies  of  profoeol  members. 

7.1  Liveness  guarantees 

There  are  fwo  disfinef  liveness  guaranfees:  wail-freedom  and  single-elienl  wail-freedom.  These  guaranlees 
hold  so  long  as  fhe  slorage  eapaeily  on  storage-nodes  is  nol  exhausted. 

Wait-freedom.  Wail-freedom  is  a  desirable  liveness  properly  [10].  Informally,  aehieving  wail-freedom 
means  fhaf  eaeh  elienf  ean  eomplele  ifs  operations  in  finilely  many  sleps  regardless  of  fhe  aelions  performed 
or  failures  experieneed  by  olher  elienfs.  For  a  formal  definilions  see  [10]. 

Single-client  wait-freedom.  In  non-repair  profoeol  members,  wail-freedom  is  nol  aehieved.  This  is  beeause 
read  operations  may  aborf  due  lo  eoneurreney  or  fhe  failure  of  olher  elienfs.  The  sfrongesl  slalemenf  aboul 
fhe  liveness  of  non-repair  member  proloeols  is  fhaf  a  single  eorreel  elienf  is  wail-free.  I.e.,  in  a  syslem  whieh 
is  eomprised  of  a  single  eorreel  elienf  fhaf  performs  operalions  sequenlially,  all  operations  are  wail-free.  The 
wrile  operations  of  all  profoeol  members  are  wail-free;  if  is  only  fhe  read  operalion  for  whieh  fhe  weaker 
single-elienl  wail-freedom  is  required. 

Unbounded  storage  capacity.  In  fhe  proof  of  liveness  for  read  operations,  we  assume  fhaf  storage-nodes 
have  unbounded  storage  eapaeily  (i.e.,  lhal  Ihe  entire  version  history  baek  to  Ihe  initial  value  _L  al  time  0 
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is  available  at  each  storage-node).  To  prevent  capacity  exhaustion,  some  garbage  collection  mechanism  is 
required.  Garbage  collection  reduces  the  liveness  of  read  operations.  A  read  operation  that  is  concurrent  to 
write  operations  and  to  garbage  collection  may  not  observe  a  complete  candidate.  The  read  operation  can 
observe  a  series  of  incomplete  candidates  that  complete  and  are  garbage  collected  within  the  duration  of  the 
read  operation.  In  such  a  situation,  the  read  operation  would  observe  _L  at  some  timestamp  other  than  0  from 
storage-nodes,  indicating  that  the  client  has  “skipped”  over  a  complete  write  operation.  The  read  operation 
then  must  be  retried.  The  implementation  details  of  garbage  collection  and  its  impact  on  liveness  properties 
is  given  in  [8]. 

7.2  Proof 

All  liveness  properties  hinge  on  the  following  lemma. 

Lemma  5  All  operations  eventually  receive  at  least  N  —  t  responses. 

Proof:  In  the  crash-recovery  model,  there  are  at  least  N  —  t  good  storage-nodes  (i.e.,  storage-nodes 
that  are  always-up  or  eventually-up).  By  definition,  eventually,  all  good  storage-nodes  will  be  up.  Since 
all  requests  to  storage-nodes,  from  clients,  are  retried  until  N  —  t  responses  are  received,  eventually,  N  —  t 
responses  will  be  received  (see  READ  .TIMESTAMP,  DO.WRITE,  and  DO  JlEAD).  □ 

Observation  6  It  is  possible  for  progress  to  be  made  throughout  the  duration  of  a  run,  not  just  once  all 
good  storage-nodes  are  up.  Lemma  5  guarantees  that  eventually  N  —  t  responses  will  be  received.  During 
any  period  in  which  N  —  t  storage-nodes  are  up,  operations  may  receive  N  —  t  responses  and  thus  complete. 
In  fact,  responses  can  be  collected,  over  time,  from  N  —  t  storage-nodes,  during  a  period  in  which  fewer  than 
N  —  t  storage-nodes  are  ever  up  (but  during  which  some  storage-nodes  crash  and  some  recover). 

7.2.1  Asynchronous  repairable 

The  asynchronous  repairable  protocol  member  provides  a  strong  liveness  property,  namely  wait-freedom  [  10, 
14].  Informally,  each  operation  by  a  correct  client  completes  with  certainty,  even  if  all  other  clients  fail,  pro¬ 
vided  that  at  most  b  servers  suffer  Byzantine  failures  and  no  more  than  t  servers  are  not  good. 

Lemma  6  A  write  operation  by  a  correct  client  completes. 

Proof.  A  write  operation  by  a  correct  client  waits  for  A  —  f  responses  from  storage-nodes  before  re¬ 
turning.  By  Lemma  5,N  —  t  responses  can  always  be  collected.  Since,  Qc<N  —  t  —  b  (cf.  (4)  in  Section  5) 
for  repairable  protocol  members,  then  N  —  t  >Qc  +  b.  Since  at  most  b  storage-nodes  are  Byzantine,  then  at 
least  Qc  benign  storage-nodes  execute  write  requests,  which  completes  the  write  operation.  □ 

Lemma  7  A  read  operation  by  a  correct  client  completes. 

Proof:  Given  N  —  t  READ JtESPONSE  messages,  a  read  operation  classifies  a  candidafe  as  complefe, 
repairable,  or  incomplete.  The  read  complefes  if  a  candidafe  is  classified  as  complete.  As  well,  fhe  read 
complefes  if  a  candidafe  is  repairable.  Repair  is  initialed  for  repairable  candidafes — ^repair  performs  a  wrife 
operafion,  which  by  Lemma  6  completes — which  lefs  fhe  read  operafion  complete.  In  fhe  case  of  an  incom- 
plefe,  fhe  read  operafion  Iraverses  fhe  version  history  backwards,  until  a  complefe  or  repairable  candidafe  is 
discovered.  Traversal  of  fhe  version  hisfory  ferminafes  if  J_  af  logical  time  0  is  encounfered  af  Qc  storage- 
nodes.  □ 
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7.2.2  Asynchronous  non-repair 

Like  the  asynchronous  repairable  member,  write  operations  of  the  asynchronous  non-repair  member  are 
wait-free.  Indeed,  Lemma  6  holds,  since  Qc  <N  —  2t  —  2b<N  —  t  —  bfor  the  asynchronous  non-repair 
member  (cf.  (7)  in  Section  5). 

Read  operations  of  the  asynchronous  non-repair  member  may  abort  (i.e.,  return  _L  at  some  time  greater 
than  0).  However,  we  can  make  a  limited  statement  about  the  liveness  of  read  operations. 

Lemma  %  In  a  system  comprised  of  a  single  correct  client,  all  read  operations  complete. 

Proof:  Write  operations  by  the  correct  client  are  executed  by  at  least  N  —  t  —  b  benign  storage-nodes. 
Since  Qc^N  —  2t  —  2b,  then  at  least  Qc  +  boi  the  benign  storage-nodes  that  execute  the  write  operation  are 
in  the  candidate  set  of  a  subsequent  read  operation.  A  candidate  set  of  size  Qc  ^  is  classified  as  complete. 
Therefore,  read  operations  complete.  □ 

8  Synchronous  protocol  family  members 

In  this  section  we  consider  the  constraints  on  N,  Qc,  and  m  for  synchronous  members  of  the  protocol 
family.  We  consider  the  constraints  under  three  related  failure  models:  crash-recovery,  omission,  and  fail- 
stop.  The  crash-recovery  failure  model  is  a  strict  generalization  of  the  omission  and  fail-stop  failure  models. 
The  omission  failure  model  is  a  strict  generalization  of  the  fail-stop  failure  model.  Synchronous  protocol 
members  differ  from  asynchronous  in  that  there  are  distinct  constraints  for  each  failure  model.  The  lower 
bound  on  N  decreases  as  the  storage-node  failures  tolerated  become  less  general.  We  maintain  a  hybrid 
failure  model,  in  that  b  storage-nodes  may  be  Byzantine,  and  t  >b  storage-nodes  may  have  omissions  (or, 
may  fail-stop). 

8.1  Crash- recovery 

In  a  synchronous  protocol  member,  it  is  possible  to  wait  for  responses  from  all  storage-nodes  (where 
TIMEOUT  may  be  the  response).  Thus,  if  more  than  N  —  t  storage-nodes  are  up,  the  read  classification  rule 
has  more  information  with  which  to  classify  incomplete  candidates.  To  classify  a  candidate  as  incomplete, 
the  candidate  must  be  incomplete  (i.e.,  fewer  than  Qc  benign  storage-nodes  have  executed  the  write).  Let 
/  be  the  number  of  storage-nodes  that  have  timed  out.  Since  operations  retry  requests  until  N  —  t  responses 
are  received,  0  <  /  <  f.  In  synchronous  members,  the  rule  for  classifying  a  candidate  incomplete  is, 

I CandidateSet\  +  f  <  Qc,  so  INCOMPLETE  =  Qc-f,  (9) 

Basically,  each  storage-node  that  has  timed  out  must  be  assumed  to  be  benign,  and  to  have  executed  the 
write  operation. 

In  the  case  that  t  storage-nodes  timeout  in  a  system,  then  (9)  is  identical  to  (3).  However,  if  fewer  than 
t  storage-nodes  timeout,  then  (9)  has  more  information  with  which  to  classify  candidates  as  incomplete. 

The  ability  to  better  classify  incomplete  candidates  is  the  major  difference  between  asynchronous  and 
synchronous  protocol  members  in  the  crash-recovery  model.  Specifically,  the  constraints  on  N,  Qc,  and  m, 
listed  in  Table  2  apply  to  the  synchronous  crash-recovery  protocol  members.  As  well,  the  safety  proof  given 
in  Section  6  and  liveness  proofs  given  in  Section  7  applies. 
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Protocol 

Omission  repairable 

Omission  non-repair 

Fail-stop  repairable 

Fail-stop  non-repair 

N 

2t+l<N 

2t  +  b+l<N 

t  +  b+l<N 

t  +  2b+l<N 

Qc 

t  +  l<Qc 

Qc<N~t 

t  +  l<Qc 

Qc<N-t-b 

t  +  l<Qc^ 

Ql^  <N-b 

t  +  l<Qc^ 

Qc^  <N~2b 

m 

1  <  m  <  Qc  —  t 

1  <m<  Qc 

1  <m<  2^®  —  t 

1  <  m  <  Qq^  —t  +  b 

Table  3:  Synchronous  protocol  family  members  constraint  summary 


DO_WRITE({Z)|,...,Z)]v},  ts,  CC)  : 

1:  for  all  S,  G  {5i, . . .  ,5iv}  do 
2:  SEND(5,,  WRITE-REQUEST,  ts,  Z),,  CC) 

3 :  end  for 
4:  ResponseSet  :=  0 
5:  repeat 

6:  ResponseSet  :=  ResponseSet  U  RECEIVE_WRITEJlESPONSE(S) 

7:  until  {\ResponseSet\  =W  OR  TIMEOUT) 

Figure  4:  Synchronous  implementation  of  D0_WRITE  with  reliable  channels. 

8.2  Less  general  failure  models 

A  distinguishing  feature  of  the  omission  and  fail-stop  failure  models,  eompared  to  the  erash-reeovery  model, 
is  that  the  loeations  of  failures  are  fixed.  That  is,  A  —  f  storage-nodes  are  always-up.  There  are  no  storage- 
nodes  that  are  eventually-up — no  more  than  t  storage-nodes  may  experience  failures  of  any  kind. 

The  constraints  on  N,  m,  and  Qc  for  synchronous  protocol  members  under  the  omission  and  fail-stop 
failure  models  are  given  in  Table  3.  Notice  that  the  definition  of  Qc  is  modified  for  fhe  fail-sfop  failure 
model  (fhus,  fhe  ferm  2^^). 

8.2.1  Reliable  channels 

Under  fhe  omission  and  fail-slop  failure  models,  we  assume  reliable  channels.  Under  fhe  crash-recovery 
model,  we  implemenl  reliable  channels  by  repealedly  sending  requesls  fo  sefs  of  slorage-nodes  until  suf- 
ficienl  responses  are  received.  If  is  easier  lo  Ihink  aboul  fhe  synchronous  omission  and  fail-slop  prolocol 
members  by  assuming  reliable  channels. 

The  assumplion  of  reliable  channels  affecls  fhe  implemenfalion  of  fhe  funcfions  READ_TIMESTAMP, 
DO.WRITE,  and  DOJtEAD.  The  change  of  implemenfalion  is  Irivial.  We  illuslrale  fhe  modified  implemenla- 
lion  wilh  fhe  funclion  D0_WRITE  in  Figure  4.  The  functions  READ_TIMESTAMP  and  DOJtEAD  are  similarly 
modified. 

Since  fhe  channels  are  assumed  lo  be  reliable,  messages  lo  fhe  slorage-nodes  are  senl  jusl  once  (cf. 
line  2).  Responses  are  collecled  until  a  lolal  of  N  responses  are  collecled,  or  until  fhe  TIMEOUT  for  fhe 
synchronous  sysfem  is  reached  (cf.  line  7).  If  fhe  loop  exils  because  TIMEOUT  is  reached,  by  assumplion, 
fhe  response  sel  has  al  leasl  N  —  t  members.  Slricfly  speaking,  fhe  second  half  of  fhe  function  which  collecls 
responses  lo  Ihe  write  requesls  is  unnecessary.  Il  may  be  useful  for  Ihe  clienl  lo  know  which  slorage- 
nodes  acknowledge  Ihe  write  requesls,  and  which  have  timed  oul;  however,  because  of  Ihe  reliable  channels, 
D0_WRITE  could  relurn  after  line  3. 

8.3  Omission  failure  model 

A  slorage-node  lhal  experiences  an  omission  failure  eilher  does  nol  receive,  or  does  nol  send  a  message  [21]. 
A  clienl  lhal  “observes”  an  omission,  in  lhal  Ihe  slorage-node  did  nol  reply  wilhin  Ihe  synchronous  bound 
of  channel  delay  and  processing  delay,  receives  a  limeoul.  Given  lhal  no  more  lhan  t  slorage-nodes  may 
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fail,  a  client  is  guaranteed  to  receive  N  responses,  no  more  than  t  of  which  are  timeouts.  If  a  client  receives 
a  timeout  from  some  storage-node,  then  some  other  client,  in  a  subsequent  request,  may  receive  a  response 
from  the  storage-node.  This  is  the  nature  of  omission  failures. 

Under  the  omission  failure  model,  the  definition  of  a  complete  write  operation  is  the  same  as  in  the 
crash-recovery  model.  However,  the  classification  rule  for  a  complete  candidate  is  modified  fo  fake  advan- 
fage  of  observed  failures.  Defining  /,  as  above,  as  fhe  number  of  fimeoufs  received,  if  f  >  t  —  b,  fhen  fhere 
are  af  mosf  t  —f  Byzantine  sforage-nodes  in  fhe  candidafe  sef.  For  example,  if  f  =  t,  fhen  all  responses  re¬ 
ceived,  fhaf  are  nof  fimeoufs,  are  from  correcf  sforage-nodes.  To  reason  abouf  fhis  abilify  in  fhe  consfrainfs, 
we  infroduce  b,  which  is  defined  as  follows: 


^  (b  iff<t-b- 
\  t-f  if f>t-b. 


(10) 


In  a  synchronous  model  wifh  omission  failures,  fhe  complefe  classification  rule  is, 

\CandidateSet\  —b>  Qc,  so  COMPLETE  =  Qc  -h  b.  (1 1) 

The  incomplefe  classificafion  rule  is  fhe  same  as  in  fhe  crash-recovery  model, 

\CandidateSet\  +f  <Qc,  so  INCOMPLETE  =  Qc-f-  (12) 


The  more  responses  received  by  a  clienf,  fhe  “beffer”  if  can  classify  incomplefe  candidafes,  since  /  is  lower. 


8.3.1  Repairable  protocol  member  constraints 

Write  completion:  There  must  be  sufficient  correct  storage-nodes  in  the  system  for  a  write  operation  to 
complete.  Since  only  t  storage-nodes  may  fail,  and  since  correct  storage-nodes  always  reply  to  clients,  then 

Qc<N-t.  (13) 

Real  unclassifiable/repairable  candidates:  No  additional  constraints  are  necessary  to  ensure  that 
Byzantine  storage-nodes  cannot  fabricate  repairable  candidates.  Consider  (12),  the  incomplete  classification 
rule  under  omission  failures  in  synchronous  systems.  If  b  Byzantine  storage-nodes  fabricate  a  candidate, 
then  at  most  t  —  b  storage-nodes  timeout.  Substituting  f  =  t  —  b  into  (12),  we  have  | CandidateSet\  +  t  —b  < 
Qc-  Since,  \CandidateSet\  <  b,  then  this  becomes  t  <Qc  (which  is  redundant,  given  (1),  the  write  durability 
constraint).  So  long  as  the  candidate  set  has  b  or  fewer  members,  it  is  classified  as  incomplete;  therefore, 
Byzantine  storage-nodes  cannot  fabricate  a  repairable  candidate. 

Decodable  repairable  CANDIDATES:  If  t  storage-nodes  do  not  respond,  then  the  classification  rule  for 
incomplete,  (12),  leads  to  the  threshold  for  repairable  candidates, 

1  <  m  <  Qc  —  t.  (14) 

Constraint  summary: 

Qc  +  t 

f  -p  1  <  Qc  ^  N  —  t', 

1  <m<  Qc  —  t. 
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8.3.2  Non-repair  protocol  member  constraints 

The  above  real  unelassifiable  eandidate  eonstraint  holds  for  the  non-repair  protoeol  member  (i.e.,  t  <  Qc)- 
The  write  eompletion  eonstraint  is  supereeded  by  the  elassifiable  eomplete  eandidates  eonstraint. 
Classifiable  complete  candidates:  For  this  property  to  hold,  a  read  operation  must  observe  at  least 
Qc  +  b  responses  from  eorreet  storage-nodes — suffieient  responses  to  elassify  the  eandidate  as  eomplete. 
Sinee  b  may  equal  b  in  the  eomplete  elassifieation  rule,  (11),  and  sinee  a  write  operation  may  only  reeeive 
responses  from  N  —  t  storage-nodes,  then 


Qc+b<N -t, 

Qc<N-t-b.  (15) 

Decodable  complete  CANDIDATES:  A  eandidate  elassified  as  eomplete,  (11),  must  be  deeodable.  Sinee, 
if  /  =  t,  ^  =  0,  it  is  possible  to  elassify  something  as  eomplete  with  a  eandidate  set  with  only  Qc  members, 

1  <  m  <  Qc-  (16) 


Constraint  summary: 


Qc  +  t  +  b<N-, 

t  +  l  <  Qc  —  t  —  b; 
1  <m<  Qc- 


8.4  Fail-stop  failure  model 

A  storage-node  that  experienees  a  fail-stop  failure,  erashes  in  sueh  a  manner  that  it  takes  no  further  aetion 
and  its  failure  is  deteetable  [25].  In  a  synehronous  system,  this  means  that  elients  ean  deteet  that  a  storage- 
node  has  failed — although  Byzantine  storage-nodes  ean  appear  fail-stopped  to  some  elients  and  up  to  other 
elients.  Under  the  fail-stop  failure  model,  the  definition  of  a  eomplete  write  operation  is  modified  from  fhaf 
of  fhe  erash-reeovery  model  (and  omission  model). 

Complete  write  operation:  A  write  operafion  is  defined  fo  be  eomplete  onee  a  fofal  of  2^^  benign 
sforage-nodes  have  exeeufed  wrife  requesfs  or  have  fail-slopped. 

The  ehange  in  definilion  of  a  eomplele  wrife  operation  affeefs  fhe  elassifieation  rules  for  eomplefe  and 
ineomplefe  eandidates: 

\CandidateSet\  +f  —  b>  2^^,  so  COMPLETE  =  2^^  —f  +  b',  (17) 

I CandidateSet\  -f  /  <  2c^,  so  INCOMPLETE  =  2c^  -  /•  (18) 

For  fhe  eomplele  elassifieafion  rule,  if  is  assumed  lhal  up  lo  b  of  fhe  responses  eould  be  from  Byzanline 
sforage-nodes  (sueh  sforage-nodes  eould  by  lying  aboul  a  value  or  prelending  fo  be  fail-slopped).  For  fhe 
ineomplefe  elassifieation  rule,  if  is  assumed  fhaf  all  observed  fail-slop  responses  are  from  benign  sforage- 
nodes. 

8.4.1  Repairable  protocol  member  constraints 

Write  completion:  There  must  be  suffieient  benign  storage-nodes  in  the  system  for  a  write  operation  to 
eomplete.  Sinee  only  b  storage-nodes  may  be  Byzantine,  and  the  remainder  are  benign, 

Qf<N-b.  (19) 
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Real  unclassifiable/repairable  candidates:  For  reasons  similar  to  the  omission  failure  model,  the 
write  durability  eonstraint  {t  <  2^^)  is  suffieient  for  ensuring  that  Byzantine  storage-nodes  eannot  fabrieate 
repairable  eandidates.  Speeifieally,  if  b  Byzantine  storage-nodes  fabrieate  a  eandidate,  then  at  most  t  —  b 
storage-nodes  fail-stop,  so  from  (18),  we  have  \CandidateSet\  +  t  —  b  <  2^^.  And,  sinee  2^^  >  t,  we  have, 
\CandidateSet\  <  b.  Therefore  Byzantine  storage-nodes  eannot  fabrieate  a  repairable  eandidate. 

Decodable  repairable  CANDIDATES:  Given  f  =  t  failures,  an  operation  ean  be  elassified  as  repairable 
with  as  few  as  2^^  —  t  members  in  the  eandidate  set,  thus, 

\<m<Ql^-t.  (20) 


Constraint  summary: 


Qf+b<N- 

t  +  l<Qf  <N-b; 
\  <m<Ql^ -t. 


8.4.2  Non-repair  protocol  member  constraints 

The  bound  on  the  above  write  eompletion  eonstraint  is  supereeded  by  the  elassifiable  eomplete  eandidates 
eonstraint. 

Classifiable  complete  candidates:  For  this  property  to  hold,  a  read  operation  must  observe  at  least 
—  f  +  b  responses  that  mateh.  Considering  the  ease  of  /  =  0  leads  to  the  lower  bound  on  N.  If  /  = 
0,  then  2^  -|-  b  responses  must  mateh.  Sinee  up  to  b  storage-nodes  ean  lie,  then  there  must  be  at  least 
Qf+b  +  b  <  N  storage-nodes  to  ensure  this  property. 

Qf+b  +  b<N, 

Qf<N-lb.  (21) 

Decodable  complete  candidates:  A  eandidate  elassified  as  eomplete,  (17),  must  be  deeodable.  Sinee, 
/  may  be  as  great  as  t, 

l<m<Q^(^ -t+b.  (22) 


Constraint  summary: 


Qf  +  lb<N- 

t  +  l<  Qf  <N-lb\ 

\  <m<  2^^  —t+b. 

9  Synchronous  members  with  synchronized  clocks 

Protoeols  to  aehieve  approximate  eloek  synehronization  in  today’s  networks  are  well  known,  inexpensive, 
and  widely  deployed  [20].  In  this  seetion,  we  eonsider  the  use  of  synehronized  eloeks  for  synehronous  pro- 
toeol  family  members.  Synehronized  eloeks  and  weakening  safety  allow  the  write  operation  to  be  performed 
in  a  single  phase,  rather  than  in  two  phases.  Speeifieally,  there  is  no  need  to  determine  the  “eurrent”  logieal 
time  by  sending  requests  to  storage-nodes;  the  loeal  eloek  time  is  used.  Thus,  line  2  of  the  funetion  WRITE 
in  Figure  2  beeomes  Time  :=  READ_L0CAL_CL0CK(),  and  the  funetion  READ  .TIMESTAMP  is  not  needed. 
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9.1  Clock  skew 

The  use  of  elient  eloek  synehronization  introduees  eloek  skew  (i.e.,  eloeks  are  not  perfeetly  synehronized — 
they  are  synehronized  to  within  some  bound,  the  skew).  Sinee  there  may  be  skew  among  elient  eloeks, 
the  definition  of  operation  duration  must  be  modified;  fhis  weakens  fhe  safefy  guaranfee.  Speeifieally,  we 
exfend  fhe  definition  of  when  a  wrife  operafion  begins,  fo  aeeommodafe  fhaf,  wifhin  fhe  eloek  skew,  wrife 
operafions  may  be  linearized  “ouf  of  order”. 

From  Lemma  2,  we  observed  fhaf  fhe  fimesfamp  order  is  a  fofal  order  on  wrife  operafions.  We  define 
X  fo  be  fhe  maximum  skew  befween  fhe  eloeks  of  any  fwo  eorreef  elienfs.  For  synehronous  members  wifh 
synehronized  eloeks,  we  redefine  when  a  benign  elienf  begins  a  wrife  operafion  (ef.  Detinifion  5): 

Definition  8  Wts  begins  x  before  a  benign  elienf  invokes  fhe  WRITE  operafion  loeally  fhaf  issues  a  wrife 
requesf  bearing  fimesfamp  ts. 

Consider  fwo  elienfs,  A  and  B,  whose  eloeks  differ  in  fhaf  A’s  eloek  is  X  less  fhan  B’s  eloek.  If  is 
neeessary  fo  exfend  fhe  begin  time  of  a  wrife  operafion  fo  aeeommodafe  fhe  ease  when  B  invokes  a  wrife 
operafion  less  fhan  x  before  A  invokes  a  wrife  operafion.  If  B  exeeufes  write  requesfs  af  Qc  benign  storage- 
nodes  before  A  invokes  ifs  write  operafion,  fhen  B’s  wrife  operafion  is  eomplefe  when  A’s  wrife  operafion 
begins.  However,  A’s  wrife  operafion  will  be  linearized  before  B’s  write  operafion — if  is  linearized  “ouf  of 
order”. 

9.2  Byzantine  clients 

Using  purely  logieal  time,  fhe  begin  limes  of  write  operafions  by  Byzanline  elienfs  is  undefined.  As  dis- 
eussed  in  Seelion  6.1,  Ibis  allows  sueh  wrife  operations  fo  be  linearized  jusf  prior  to  some  ofher  wrife 
operafion  sueh  fhaf  no  read  operation  observes  Ihem  (i.e.,  sueh  fhaf  Ihey  have  no  effeel).  Wrife  operafions  by 
Byzanline  elienfs  fhaf  are  “baek-in-lime”  in  synehronous  members  wifh  synehronized  eloeks  ean  be  Irealed 
similarly. 

Byzanline  elienfs  fhaf  wrife  “into  fhe  fulure”  require  additional  logie  during  fhe  read  operation  fo  handle 
eorreelly.  A  eorreef  elienf  musl  ignore  (elassify  as  ineomplele)  eandidales  wifh  a  fimesfamp  grealer  fhan  x 
in  fhe  fulure  relalive  to  ifs  loeal  eloek.  No  eorreef  elienf  ean  perform  a  wrife  operafion  whieh  a  subsequenl 
read  operation  by  a  eorreef  elienf  observes  as  being  grealer  fhan  x  in  fhe  fulure  relative  fo  ifs  loeal  eloek. 

If  slorage-nodes,  as  well  as  elienfs,  have  synehronized  eloeks,  fhen  slorage-nodes  ean  rejeel  wrife 
requesfs  fhaf  are  “info  fhe  fulure”  (i.e.,  grealer  fhan  x  in  fhe  fulure  relalive  to  ifs  loeal  eloek).  Given  slorage- 
nodes  wifh  synehronized  eloeks,  no  addifional  elienf  logie  is  neeessary  on  a  read  operafion;  Byzanline  elienfs 
eannol  wrife  “info  fhe  fulure”. 

Moreover,  slorage-nodes  eould  rejeel  “baek  in  lime”  write  requesfs.  The  bound  on  Iransmission  delay 
and  proeessing  time  is  used  fo  define  a  “baek  in  lime”  wrife  requesf.  Lei  (|)  be  fhe  maximum  delay  due  to 
message  Iransmission,  sender  proeessing  afler  reading  ifs  loeal  eloek,  and  reeeiver  proeessing  before  reading 
ifs  loeal  eloek.  Thus,  a  slorage-node  ean  rejeel  a  “baek  in  lime”  wrife  requesf  if  fhe  fimesfamp  of  fhe  requesf 
is  more  fhan  x  -|-  (|)  in  fhe  pasl  relalive  fo  ifs  loeal  eloek.  By  assumpfion/detinilion,  only  Byzantine  elienfs  ean 
issue  wrife  requesfs  whieh  eorreef  slorage-nodes  rejeel.  Note  fhaf  a  slorage-node  wifh  a  eloek  fhaf  is  ouf  of 
synehronization  is  eonsidered  fo  be  Byzanline. 

10  Related  work 

We  review  work  related  fo  fhe  eoneepl  of  proloeol  families,  aeeess  proloeols,  and  survivable  storage  syslems 
(ineluding  quorum  syslems  and  erasure-eoded  syslems)  in  [9].  Here,  we  foeus  on  work  related  fo  fhe  failure 
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models  employed  and  the  subsequent  safety  and  liveness  properties  aehieved  by  members  of  the  protoeol 
family  (espeeially  in  the  eontext  of  Byzantine  elients). 

We  note  that  the  protoeol  family  was  developed  in  the  eontext  of  the  BASIS  survivable  storage  sys¬ 
tem  projeet  [28,  9,  7].  Sinee  storage-nodes  aetually  have  finite  eapaeity,  a  garbage  eolleetion  meehanism 
is  needed.  Diseussion  of  some  implementation  details  of  garbage  eolleetion  and  its  impaet  on  liveness 
properties  is  given  in  [8]. 

10.1  Failure  models 

We  eonsider  many  failure  models.  We  make  use  of  a  hybrid  failure  model  for  storage-nodes  [27].  However, 
we  move  beyond  a  mix  of  erash  and  Byzantine  failures  to  allow  for  a  mix  of  erash-reeovery,  omission  or 
fail-stop  failures  and  Byzantine  failures.  The  erash-reeovery  model  was  introdueed  by  Aguilera,  Chen,  and 
Toueg  [1].  The  omission  failure  model  was  introdueed  by  Perry  and  Toueg  in  [21].  The  fail-stop  failure 
model  was  introdueed  by  Sehliehting  and  Sehneider  [24,  25].  The  Byzantine  failure  model  was  introdueed 
by  Lamport,  Shostak,  and  Pease  [16].  Baekes  and  Caehin  have  also  eonsidered  the  “hybridization”  of 
Byzantine  faults  with  a  erash-reeovery  model  for  reliable  broadeast  [3]. 

The  protoeol  family  we  have  developed  is  not  adaptive  with  regard  to  the  faults  tolerated — eaeh  family 
member  tolerates  a  statie  failure  model.  This  is  in  elear  eontrast  to  work  by  Chen,  Hiltunen,  and  Sehlieht¬ 
ing  [13,  5]  in  whieh  (fault-tolerant)  systems  are  developed  that  graeefully  adapt  to  ehanges  in  the  exeeution 
environment  or  user  requirements  by  switehing  the  protoeols  employed.  Adaptive  teehniques  for  Byzan¬ 
tine  quorum  systems  were  developed  by  Alvisi,  Malkhi,  Pieree,  Reiter,  and  Wright  [2].  The  applieation 
of  adaptive  fault  thresholds  to  Byzantine  quorum  systems  eould  inform  future  extensions  of  our  protoeol 
family. 

10.2  Safety,  liveness,  and  Byzantine  clients 

To  provide  reasonable  storage  semanties  a  system  must  guarantee  that  readers  see  eonsistent  answers.  For 
shared  storage  systems,  this  usually  means  linearizability  [12]  of  operations.  Jayanti  refined  the  notion  of 
wait-freedom  to  address  fault-tolerant  shared  objeets  [14]. 

Although  we  foeus  on  aehieving  wait-freedom,  we  also  had  to  eonsider  what  we  ealled  single-elient 
wait-freedom  (see  Seetion  6.1).  It  is  similar  to,  but  weaker  than,  obstruetion-freedom  [11]  by  Herlihy, 
Luehango,  and  Moir.  Obstruetion-freedom  guarantees  progress  onee  eoneurreney  subsides.  Single-elient 
wait-freedom  guarantees  progress  onee  eoneurreney  subsides  only  if  all  elients  are  eorreet  (assuming  elients 
retry  if  a  read  operation  aborts  in  non-repair  protoeol  members). 

In  synehronous  fail-stop  protoeol  members,  the  safety  and  liveness  guarantees  are  not  pure.  Charron- 
bost,  Toug,  and  Basu  identify  safety  and  liveness  properties  that  are  not  pure  in  [4].  Sueh  properties  ean 
become  true  because  of  failures.  Our  definition  of  a  complete  write  operation  for  synchronous  fail-stop  pro¬ 
tocol  members  is  clearly  not  pure:  We  include  benign  storage-nodes  that  have  fail-stopped  in  the  definition 
of  2^  (see  Section  8.4). 

Pierce  extended  linearizability  to  include  the  possibility  of  read  aborts:  pseudo-regular  semantics  [22]. 
If  a  reader  sees  either  a  consistent  answer  or  aborts,  it  achieves  pseudo-regular  semantics.  Trivial  solutions 
(i.e.,  readers  always  aborting)  are  excluded  by  identifying  specific  conditions  under  which  a  read  operafion 
musf  refurn  a  value.  The  liveness  guarantee  of  protocol  members  fhaf  allow  aborfs,  linearizabilify  wifh  read 
aborfs,  is  very  similar  to  Pierce’s  pseudo-regular  semanfics. 

Wrife  operafions  in  fhe  protocol  family  by  Byzanfine  clienfs  do  nol  have  a  well-defined  sfarf  time 
and  are  fhus  concurrenf  to  all  preceding  operations.  Members  of  fhe  profocol  family  linearize  wrifes  by 
Byzanfine  clienfs,  if  fhey  are  “back  in  time”,  jusf  prior  to  some  ofher  wrife  operafion  so  fhaf  fhey  have 
no  effecf.  Ofher  work  does  nof  direcfly  discuss  Byzanfine  clienfs  fhaf  wrife  “back  in  time”.  Malkhi  and 
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Reiter  in  [18]  use  an  “eeho”  protoeol  to  ensure  that  write  operations  are  well-formed  (“justified”  in  their 
terminology).  The  eeho  protoeol  requires  an  additional  phase — indeed,  the  protoeol  employs  three  phases: 
get  time,  propose  (eeho),  and  commit.  The  benefit  of  the  echo  protocol  is  two-fold.  First,  it  appears  that 
Byzantine  clients  could  be  prevented  from  writing  “back  in  time”,  since  the  propose  message  includes  signed 
get  time  responses.  However  the  server  logic  presented  in  [18]  does  not  directly  address  “back  in  time” 
writes.  Indeed,  the  prior  work  of  Malkhi  and  Reiter  in  [17],  indicates  that  “back  in  time”  write  operations  by 
Byzantine  clients  are  treated  in  a  manner  similar  to  the  protocol  family  (Section  3.2  of  [17]  states  that  each 
server  modifies  ifs  value  and  timesfamp  only  if  the  timestamp  received  is  greater  than  the  latest  timestamp 
received — i.e.,  “back  in  time”  writes  are  treated  as  if  they  have  no  effect).  Second,  the  echo  phase  ensures 
that  a  Byzantine  client  is  sending  the  same  value  to  each  server.  Adding  an  echo  phase  to  the  protocol  family 
could  allow  the  begin  time  of  write  operations  to  be  defined,  but  would  not  achieve  the  second  benefit,  since 
the  protocol  family  allows  values  to  be  erasure-coded. 

Martin,  Alvisi,  and  Dahlin  in  [19]  use  a  different  approach  for  dealing  with  Byzantine  clients  in  the 
Minimal  Byzantine  Storage  protocol.  To  deal  with  Byzantine  clients,  Martin  et  al.  depart  from  the  traditional 
quorum  communication  pattern  and  allow  inter-server  communication.  Whenever  a  correct  server  receives 
a  write  request,  it  stores  the  value  and  then  broadcasts  the  value  to  other  servers.  Treating  the  data  as  the  low 
bits  of  the  timestamp,  all  correct  servers  can  “agree”  on  the  latest  value  written,  even  if  a  Byzantine  client 
sends  different  data  with  the  same  timestamp  to  each  server.  Again,  Byzantine  clients  writing  “back  in  time” 
is  not  directly  discussed  (in  terms  of  its  ramifications  on  linearizability).  However,  the  protocol  appears  to 
guarantee  a  similar  property  to  the  protocol  family:  “back  in  time”  writes  have  no  effect. 

Linearizability  is  not  well  defined  in  the  context  of  Byzantine  clients.  We  believe  that  there  may  be 
many  useful  variations  of  linearizability  with  Byzantine  clients.  Unfortunately,  the  useful  variations  may 
depend  on  the  approach  taken  by  the  protocol. 
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